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ABSTRACT 

Companies and universities are increasingly moving to online 
delivery for much of their training and education needs, and designing and 
building quality distance education is a challenge facing many organizations. 
Option Six is an independent company that is building customized e-learning 
solutions. Over the last 2 years, the instructional designers and user 
experience analysts at Option Six worked to help develop a 4-stage process 
for evaluating e-learning courses. The process is built around Donald 
Norman's (2000) definition of human-centered design. Human-centered 
development utilizes input from students in the target audience during the 
earliest stages of development. By having students involved early, 
unnecessary complexity can be eliminated and learning can be maximized. This 
paper describes the process used by Option Six and outlines the benefits and 
challenge of human-centered design for distance education. (Author) 
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Designing and building quality distance education is a challenge facing many organizations. Option Six is 
an independent company building customized e-learning solutions. Over the last two years, the 
instructional designers and user experience analysts at Option Six worked to help develop a four-stage 
process for evaluating e-learning courses. The process is built around Donald Norman ’s (2000) definition 
of human-centered design. This paper describes the process used by Option Six and outlines the benefits 
and challenge of human-centered design for distance education. 

Introduction 



Companies and universities are increasingly moving to online delivery for much of their training and 
education needs. Start-ups, consulting companies, and even universities are touting their ability to deliver quality e- 
leaming to employees worldwide. With so many entities vying for students on a global scale, the issue of quality is 
becoming an increasing concern. Evaluating distance education is problematic in a number of ways. First, 
traditional student evaluations at the end of a face-to-face course typically focus on the experience students have 
with a particular instructor. With most online education, these methods are not valid as a single measure because the 
instructor’s role is only one part of a much broader experience. Technology, the user interface, and the design of 
content are all keys in understanding the distance learner’s experience. 

Second, in a face-to-face course, the instructor has the benefit of being able to see the students’ reactions to 
a number of varied teaching exercises. Nodding heads, yawns, and verbal interaction all serve as tacit “formative” 
evaluation of a course in progress. Distance courses do not have such luxury and not having that interaction can 
cause severe problems for the students and the instructor (Wang-Chavez & Branon 2000). 

Finally, the rapid pace of change in e-leaming makes knowing exactly which design principles are 
“correct” difficult to determine (Williams, Paprock, & Covington, 1999). Donald Norman President of UNext 
Learning Systems and author of such books as “The Design of Every Day Things” and “The Invisible Computer”, 
has noted that computers are complex and hard to use because we are asking them to do complex tasks (Norman, 
2000). Few tasks are as complex as education and finding the “best way” to design online education is likely to be 
an elusive goal for quite some time. Human-centered development, however, utilizes input from students in the 
target audience during the earliest stages of development. By having students involved early, unnecessary 
complexity can be eliminated and learning can be maximized. 

Option Six 






Located in Bloomington, IN, Option Six opened in January 2000 as the Bloomington Development Center 
(BDC) for UNext, Incorporated. In September 2001 the BDC spun off to form a new company, building and 
designing online courses for a number organizations. The team at Option Six has built courses with faculty from 
Stanford, Columbia, Carnegie -Mellon, The London School of Economics, the University of Chicago, and Indiana 
University. Students at a variety of organizations including General Motors, Merrill Lynch, UCLA, and MIT have 
taken courses built by Option Six team members. 

In order to create high-quality courses. Option Six utilizes a team-based approach driven by human- 
centered design principles. Each team is comprised of members from a variety of disciplines. The point-person for 
the team is an instructional designer. Instructional designers work with subject-matter experts (SME) to determine 
the pedagogical strategy for the course. Editors set the tone for the course and ensure consistency in language usage. 
Visual designers create the look and feel for a course, including graphics, and work closely with multimedia 



345 



2 



developers to build interactive components (Flash, video, etc.). User Experience (UE) analysts work with the team 
to gather data from students in the target audience, which is the core of the human-centered design process. 

Human-Centered Design Process 

Donald Norman, former President of UNext Learning Systems, defines human-centered design in his book, 
The Invisible Computer: 

“It’s a process of product development that starts with users and their needs rather than with the 

technology. The goal is a technology that serves the user, where the technology fits the task and 

the complexity is that of the task, not the tool.” (p.185). 

Option Six uses an online course development process that is driven by this definition. The process has 
four primary components that focus development around the student, rather than the technology. Guiding the 
development of the process is a desire to produce the highest quality online courses with cost-effective user-testing 
methods. Each phase is designed to maximize resources while minimizing the impact on the overall development 
timeline. 

Each stage in the process has a specific objective. Early stages are designed to catch major design flaws 
and to identify conceptual difficulties with the material. Large structural flaws in the interface or content tend to be 
the most expensive and time-consuming problems to correct late in the process. The difficulty is that the early 
stages of development are the most conceptual and can be somewhat difficult to test with large numbers of users. 

The result is a process that starts with expert evaluators (User Experience analysts) conducting design 
consulting. In the second phase, participatory design, two or three students test components of the course to identify 
learning issues. The third phase is an accelerated integration test in a laboratory. Finally, an instructional pilot is 
run with students working in naturalistic setting (work, home, etc.) at a pace that would be expected for the course. 
At any point during this process it is the data from the students that determines whether the course is ready for the 
next phase of development. Multiple revisions to materials are anticipated and even expected. 

Design Consulting 

Design consulting is not a stage in the process but is an ongoing element throughout development. In team 
meetings, this often means an analyst provides past student testing data to inform decisions. Most importantly, the 
analyst acts as an objective set of ‘fresh eyes’ looking at a course in early stages of development. The goal of the 
analyst in this phase is not to replace student testing but to act as a ‘user advocate’ within the design team. Once 
student testing begins, the design-consulting role includes making recommendations based on user feedback and 
interim evaluation of product improvements. Several distinct techniques are used to maximize the skills of the 
analyst in conjunction with student data. 

Cognitive Walkthrough 

Structured inspection methods are one important way for UE analysts to determine the viability of course 
organization. Cognitive walkthroughs are a common technique for usability experts and involve following a 
predetermined path through material to look for potential points of failure (Wharton, Rieman, Clayton, & Poison, 
1994). The effectiveness of cognitive walkthroughs is dependent on the expertise of the evaluator and will not 
necessarily catch all problems (John & Mashyna, 1997). As a part of a larger user-testing strategy, however, they 
are quite effective for providing an objective eye to inform the design team of major flaws in interface design. 

Heuristic Evaluation 

Though Option Six does not employ heuristic evaluations for all courses, this method has value when 
looking at new interfaces or small course components. Heuristic evaluations involve setting metrics for an analyst 
to use when assessing a course. The technique is less structured than a cognitive walkthrough and allows exploration 
of the course environment (Sears, 1997). Analysts are looking for aspects of the course that do not meet certain 
broad standards (readability, ease of understanding, technical problems, etc.) Again, such a technique by itself does 
not ensure a high-quality course but, as a first-stage pass through the material, it catches many major flaws. 

Participatory Design 
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Participatory design is the fist phase of a three-phase testing process. Components of the course are tested 
with two or three students from the intended target audience. The emphasis in this phase of testing is on high-level 
conceptual understanding of the course materials. Students work through a combination of early screen designs and 
paper prototypes of course materials and give continuous feedback to the analyst about whether concepts and 
procedures are clear and the interface design is usable. As the title implies, the user becomes a co-designer in the 
design process by providing suggestions for improvement rather than simply identifying problems. 

The timeline for a project is built around the idea that development will require multiple iterations based on 
this early student feedback. Complex content or media often undergo multiple changes in order to ensure students 
are able to effectively navigate and learn. Many of the worst problems with the material are identified and corrected 
at this point in the process, saving time and money as the project moves toward completion. 

Integration Testing 

When development is nearly complete, all of the components are assembled in electronic form and 
presented as a complete course. Six paid students, who have the same background as the intended target audience, 
work in a lab environment to take the course. In this phase, there is less emphasis on having the student interact 
with the design team. The timeframe is compressed (students work six hours per day online) and data is gathered 
through observation and interviews, rather than participatory design sessions. 

In university credit courses, an onsite instructor is also added during this phase to grade assignments and 
provide feedback to the team about whether students are meeting the learning objectives. The analyst works with 
observation, interview, and instructor data to develop a profile of overall course effectiveness. Problems with 
individual parts of the course are noted and the development team makes revisions based on the feedback. Severe 
problems (e.g., overall course structure is ineffective) are rare at this stage but if any are found, the entire course is 
retested after revisions. 

Instructional Pilot Testing 

After the development team is confident all major issues have been resolved, the completed course is 
uploaded to external web servers and readied for a ‘real world’ test. Eight students who match the target audience 
are recruited and paid to work on the course over a realistic timeframe in their own environment. The amount of 
time varies with the length of time students would be expected to spend on a course. A 30-hour course (roughly one 
university credit hour) that should take about six weeks to complete would be tested over a three-week period. This 
timeframe is still slightly compressed to meet development timelines but is much more realistic than in earlier 
phases of testing. 

A User Experience analyst gathers data from telephone interviews, online survey forms, instructor feedback 
and analysis of online student interaction. The focus is on understanding what potential issues exist for instructors 
and students as the course is released to paying students. If any development issues are discovered they are also 
corrected during this phase. Once this phase of testing is complete, the course is ready for release. 



Lessons Learned 

Using the process outlined above on nearly 30 courses has yielded some interesting lessons on taking a 
human-centered approach to online course design. Perhaps the single most important lesson was that the process 
itself involved iterative design. The initial implementation of the human-centered process was too slow and too 
expensive to be fiscally viable. The process outlined in this paper is the result of much trial and error and additional 
changes will certainly be made in the future. 

Benefits 



One lesson learned was that using a human-centered process is key in building high-quality online courses. 
Subsequent commercial trials and release of a number of courses showed that customers were very pleased with the 
quality of the courses. Most major technical problems and flaws were discovered prior to release, thus saving 
money by limiting the need for technical support. 

An early model of the process focused on student testing near the end of the development cycle. At that 
point, however, a great deal of time and effort had been put into programming, graphic design, and course structure. 
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Major problems were often discovered near the end of the timeline, which jeopardized due dates and cost more 
money to fix. By making students part of the design team (participatory design phase), most major problems are 
caught and corrected before too much effort is expended. Interestingly, there are generally only two or three 
students in this phase, which indicates that it only takes a few users to uncover major problems. 

Instructional designers and some members of the development team were initially skeptical of whether user 
testing could improve the product. One of the unintended benefits was that by seeing their designs being used by 
students, developers were able to improve their own understanding of how design could be shaped to improve online 
learning. 

Challenges 

Creating a human-centered design process involves many challenges. The goal is to ensure a quality user 
experience with all aspects of an online course. Initially, the User Experience group in the Bloomington office was 
staffed with usability and information science professionals. It became apparent that guaranteeing a quality 
experience for online learning would require more than ‘traditional’ interface design testing. Adding UE team 
members with instructional design experience and visual design knowledge broadened the group’s understanding of 
how students were interacting and whether they were learning. The first lesson learned was that having a 
multidisciplinary User Experience group is essential for evaluating online learning (Norman, 2000). 

As many organizations are discovering, creating quality e-leaming can be an expensive proposition. 

Finding students in the target audience for business courses (mostly business professionals) required that they be 
well paid for their time. For lengthy courses, the cost for testing, including facilities, UE salaries, and pay for 
students can run as high as $35,000. This cost is minimal for an organization looking to roll out a course to 
thousands of employees but most universities will not find an acceptable return on investment for this level of 
testing. In fact, the elimination of potential problems is an investment that can pay big dividends when the 
economies of scale a large enough. 

Another issue with this model is that students are paid for their time. Paid students will obviously have a 
very different motivation for completing a course than a paying student. This is a difficult problem to overcome 
when looking for people with specific skills and backgrounds. One potential solution to this problem is to offer 
businesses ‘free’ training if employees are willing to provide feedback to the design team. This approach is helpful 
but can only be used in the latter testing phases when the course is nearly complete. 

Conclusion 

While the process described in this paper is extremely expensive, there are low cost alternatives to each 
phase that can improve the quality of online courses. For example, in a university setting, design consulting can be 
accomplished by having a colleague look at paper prototypes of materials. Even if they do not have time to work 
through an entire course, students who have taken a face-to-face version of a course make great participatory design 
subjects. Simply asking a student walkthrough how they would get through an assignment will reveal many flaws in 
the initial design. 

Integration testing is the most problematic phase of the process to conduct on a budget. It is unlikely that 
most universities can afford to hire students to take a semester-long course before it is taught. One alternative, 
however, is to continue evaluating the course while it is being taught. This is done implicitly in face-to-face courses 
and must be explicit in distance courses. Sending a mid -semester survey to ask students how a class is going can 
provide data to make corrections and improve students’ perception of the course (Wang-Chavez & Branon 2000). 

Additionally, as a designer builds more courses within a particular domain, the need for testing should 
decrease. Option Six originally had a much more extensive testing process but as the development teams gained 
experience, less testing was needed to achieve the same result. Regardless of the level of project, getting input from 
students is an essential component in quality online course development. 

Building a human-centered process for online course development has provided a number of opportunities 
and challenges. After observing students taking a number of web-based courses the most important conclusion is 
that no designer can anticipate every issue. By putting students at the center of the design process, a team can 
proceed with confidence and prevent costly issues with the final product. 
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